Replace `Metadata.flags` with `Metadata.trans` #1060

mhauru · 2025-09-29T17:03:41Z

Now that the "del" flag is gone (#1058), the only flag that is ever used is "trans". Hence, no need to bother with having the Dict{String, BitVector} for Metadata.flags, and can instead have a single BitVector for Metadata.trans. EDIT: Renamed to Metadata.is_transformed.

You may wonder, given that Metadata is presumably on its way out, why bother? Two reasons:

I tried running the benchmark suite locally with VectorVarInfo, and there were some horrendous performance regressions there compared to using Metadata. Hence, we might not be about to switch over the VarNamedVector imminently.
The above experience made me wonder why there was such a performance difference, and whether the Metadata.flags field might actually be a significant cost compared to a BitVector.

My local benchmarking suggests that indeed, this makes a difference:

Before

┌───────────────────────┬───────┬─────────────┬───────────────────┬────────┬────────────────┬─────────────────┐
│                 Model │   Dim │  AD Backend │           VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├───────────────────────┼───────┼─────────────┼───────────────────┼────────┼────────────────┼─────────────────┤
│ Simple assume observe │     1 │ forwarddiff │             typed │  false │           16.0 │             1.7 │
│           Smorgasbord │   201 │ forwarddiff │             typed │  false │          790.6 │            46.1 │
│           Smorgasbord │   201 │ forwarddiff │ simple_namedtuple │   true │          382.0 │            84.3 │
│           Smorgasbord │   201 │ forwarddiff │           untyped │   true │         1431.7 │            36.0 │
│           Smorgasbord │   201 │ forwarddiff │       simple_dict │   true │        10511.1 │            21.6 │
│           Smorgasbord │   201 │ reversediff │             typed │   true │         1495.9 │            42.4 │
│           Smorgasbord │   201 │    mooncake │             typed │   true │         1637.4 │             3.4 │
│    Loop univariate 1k │  1000 │    mooncake │             typed │   true │         8635.9 │             3.2 │
│       Multivariate 1k │  1000 │    mooncake │             typed │   true │         1266.1 │             8.5 │
│   Loop univariate 10k │ 10000 │    mooncake │             typed │   true │        90116.3 │             3.2 │
│      Multivariate 10k │ 10000 │    mooncake │             typed │   true │        10364.2 │             9.7 │
│               Dynamic │    10 │    mooncake │             typed │   true │          235.0 │             5.7 │
│              Submodel │     1 │    mooncake │             typed │   true │           24.0 │             4.2 │
│                   LDA │    12 │ reversediff │             typed │   true │         1391.7 │             2.0 │
└───────────────────────┴───────┴─────────────┴───────────────────┴────────┴────────────────┴─────────────────┘

After

┌───────────────────────┬───────┬─────────────┬───────────────────┬────────┬────────────────┬─────────────────┐
│                 Model │   Dim │  AD Backend │           VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├───────────────────────┼───────┼─────────────┼───────────────────┼────────┼────────────────┼─────────────────┤
│ Simple assume observe │     1 │ forwarddiff │             typed │  false │           10.8 │             2.5 │
│           Smorgasbord │   201 │ forwarddiff │             typed │  false │          695.1 │            53.0 │
│           Smorgasbord │   201 │ forwarddiff │ simple_namedtuple │   true │          319.1 │           104.9 │
│           Smorgasbord │   201 │ forwarddiff │           untyped │   true │         1114.3 │            45.0 │
│           Smorgasbord │   201 │ forwarddiff │       simple_dict │   true │        10323.5 │            22.3 │
│           Smorgasbord │   201 │ reversediff │             typed │   true │         1190.0 │            52.4 │
│           Smorgasbord │   201 │    mooncake │             typed │   true │         1263.0 │             3.8 │
│    Loop univariate 1k │  1000 │    mooncake │             typed │   true │         5606.7 │             4.4 │
│       Multivariate 1k │  1000 │    mooncake │             typed │   true │         1236.0 │             8.7 │
│   Loop univariate 10k │ 10000 │    mooncake │             typed │   true │        63260.7 │             4.2 │
│      Multivariate 10k │ 10000 │    mooncake │             typed │   true │        11029.4 │             9.4 │
│               Dynamic │    10 │    mooncake │             typed │   true │          216.4 │             6.4 │
│              Submodel │     1 │    mooncake │             typed │   true │           19.0 │             4.6 │
│                   LDA │    12 │ reversediff │             typed │   true │         1341.4 │             2.0 │
└───────────────────────┴───────┴─────────────┴───────────────────┴────────┴────────────────┴─────────────────┘

Curious to see whether GHA benchmarks come out looking similar.

github-actions · 2025-09-29T17:05:24Z

Benchmark Report for Commit `a011dd6`

Computer Information

Julia Version 1.11.7
Commit f2b3dbda30a (2025-09-08 12:10 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × AMD EPYC 7763 64-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)

Benchmark Results

┌───────────────────────┬───────┬─────────────┬───────────────────┬────────┬────────────────┬─────────────────┐
│                 Model │   Dim │  AD Backend │           VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├───────────────────────┼───────┼─────────────┼───────────────────┼────────┼────────────────┼─────────────────┤
│ Simple assume observe │     1 │ forwarddiff │             typed │  false │            7.4 │             1.6 │
│           Smorgasbord │   201 │ forwarddiff │             typed │  false │          598.5 │            49.3 │
│           Smorgasbord │   201 │ forwarddiff │ simple_namedtuple │   true │          423.2 │            57.6 │
│           Smorgasbord │   201 │ forwarddiff │           untyped │   true │         1063.4 │            32.1 │
│           Smorgasbord │   201 │ forwarddiff │       simple_dict │   true │         6740.1 │            29.0 │
│           Smorgasbord │   201 │ reversediff │             typed │   true │          914.1 │            46.1 │
│           Smorgasbord │   201 │    mooncake │             typed │   true │          875.0 │             5.8 │
│    Loop univariate 1k │  1000 │    mooncake │             typed │   true │         4455.6 │             5.6 │
│       Multivariate 1k │  1000 │    mooncake │             typed │   true │         1020.9 │             9.3 │
│   Loop univariate 10k │ 10000 │    mooncake │             typed │   true │        51734.7 │             4.9 │
│      Multivariate 10k │ 10000 │    mooncake │             typed │   true │         8677.5 │            10.3 │
│               Dynamic │    10 │    mooncake │             typed │   true │          132.2 │            10.9 │
│              Submodel │     1 │    mooncake │             typed │   true │           10.4 │             5.6 │
│                   LDA │    12 │ reversediff │             typed │   true │          992.7 │             2.1 │
└───────────────────────┴───────┴─────────────┴───────────────────┴────────┴────────────────┴─────────────────┘

codecov · 2025-09-29T17:21:01Z

Codecov Report

❌ Patch coverage is 88.50575% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.36%. Comparing base (ec65b4f) to head (db64645).
⚠️ Report is 2 commits behind head on breaking.

Files with missing lines	Patch %	Lines
src/abstract_varinfo.jl	66.66%	3 Missing ⚠️
src/simple_varinfo.jl	78.57%	3 Missing ⚠️
src/threadsafe.jl	75.00%	2 Missing ⚠️
ext/DynamicPPLEnzymeCoreExt.jl	0.00%	1 Missing ⚠️
src/varinfo.jl	97.67%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           breaking    #1060      +/-   ##
============================================
- Coverage     82.38%   82.36%   -0.03%     
============================================
  Files            42       42              
  Lines          3820     3787      -33     
============================================
- Hits           3147     3119      -28     
+ Misses          673      668       -5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2025-09-30T08:56:45Z

DynamicPPL.jl documentation for PR #1060 is available at:
https://TuringLang.github.io/DynamicPPL.jl/previews/PR1060/

mhauru · 2025-09-30T08:58:52Z

CI benchmarks. Target branch:

┌───────────────────────┬───────┬─────────────┬───────────────────┬────────┬────────────────┬─────────────────┐
│                 Model │   Dim │  AD Backend │           VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├───────────────────────┼───────┼─────────────┼───────────────────┼────────┼────────────────┼─────────────────┤
│ Simple assume observe │     1 │ forwarddiff │             typed │  false │            8.5 │             1.6 │
│           Smorgasbord │   201 │ forwarddiff │             typed │  false │          635.2 │            43.6 │
│           Smorgasbord │   201 │ forwarddiff │ simple_namedtuple │   true │          411.8 │            52.7 │
│           Smorgasbord │   201 │ forwarddiff │           untyped │   true │         1163.6 │            29.7 │
│           Smorgasbord │   201 │ forwarddiff │       simple_dict │   true │         6444.2 │            28.6 │
│           Smorgasbord │   201 │ reversediff │             typed │   true │         1022.9 │            40.9 │
│           Smorgasbord │   201 │    mooncake │             typed │   true │          980.1 │             4.5 │
│    Loop univariate 1k │  1000 │    mooncake │             typed │   true │         5750.3 │             4.3 │
│       Multivariate 1k │  1000 │    mooncake │             typed │   true │          964.6 │             9.1 │
│   Loop univariate 10k │ 10000 │    mooncake │             typed │   true │        64679.1 │             3.9 │
│      Multivariate 10k │ 10000 │    mooncake │             typed │   true │         8179.8 │            10.3 │
│               Dynamic │    10 │    mooncake │             typed │   true │          129.7 │            11.3 │
│              Submodel │     1 │    mooncake │             typed │   true │           12.2 │             5.1 │
│                   LDA │    12 │ reversediff │             typed │   true │         1006.2 │             2.0 │
└───────────────────────┴───────┴─────────────┴───────────────────┴────────┴────────────────┴─────────────────┘

This branch:

┌───────────────────────┬───────┬─────────────┬───────────────────┬────────┬────────────────┬─────────────────┐
│                 Model │   Dim │  AD Backend │           VarInfo │ Linked │ t(eval)/t(ref) │ t(grad)/t(eval) │
├───────────────────────┼───────┼─────────────┼───────────────────┼────────┼────────────────┼─────────────────┤
│ Simple assume observe │     1 │ forwarddiff │             typed │  false │            7.4 │             1.7 │
│           Smorgasbord │   201 │ forwarddiff │             typed │  false │          597.3 │            49.0 │
│           Smorgasbord │   201 │ forwarddiff │ simple_namedtuple │   true │          422.1 │            57.4 │
│           Smorgasbord │   201 │ forwarddiff │           untyped │   true │          969.2 │            35.2 │
│           Smorgasbord │   201 │ forwarddiff │       simple_dict │   true │         6575.6 │            31.0 │
│           Smorgasbord │   201 │ reversediff │             typed │   true │          883.4 │            47.6 │
│           Smorgasbord │   201 │    mooncake │             typed │   true │          854.6 │             5.1 │
│    Loop univariate 1k │  1000 │    mooncake │             typed │   true │         4305.0 │             5.6 │
│       Multivariate 1k │  1000 │    mooncake │             typed │   true │          991.4 │             9.5 │
│   Loop univariate 10k │ 10000 │    mooncake │             typed │   true │        50138.4 │             5.1 │
│      Multivariate 10k │ 10000 │    mooncake │             typed │   true │         9003.3 │            10.1 │
│               Dynamic │    10 │    mooncake │             typed │   true │          128.2 │            11.4 │
│              Submodel │     1 │    mooncake │             typed │   true │            9.9 │             5.9 │
│                   LDA │    12 │ reversediff │             typed │   true │          989.8 │             2.1 │
└───────────────────────┴───────┴─────────────┴───────────────────┴────────┴────────────────┴─────────────────┘

Roughly in line with what I saw locally. Seems worth it to me, especially if you look at the Loop univariate 1k and 10k models.

yebai · 2025-09-30T11:50:53Z

I suggest we take this chance to rename Metadata.trans to a more readable term, e.g., Metadata.is_unconstrained / Metadata.is_transformed.

mhauru · 2025-09-30T14:17:01Z

Good idea, done.

src/varnamedvector.jl

penelopeysm · 2025-09-30T16:29:34Z

src/simple_varinfo.jl

-islinked(vi::SimpleVarInfo) = istrans(vi)
+islinked(vi::SimpleVarInfo) = is_transformed(vi)


like this line is just the same function but duplicated. so it feels like to me we could just pick one and roll with it!

penelopeysm

Happy to merge after this! Very pleased with the improved performance

HISTORY.md

penelopeysm · 2025-10-09T17:11:10Z

@mhauru I'll leave you to click the green button :)

Replace Medata.flags with Metadata.trans

756cc25

github-actions bot assigned mhauru Sep 29, 2025

Fix a bug

1091986

mhauru added 2 commits September 30, 2025 12:21

Fix a typo

ece8fb5

Fix two bugs

a011dd6

Rename trans to is_transformed

0f8c9b1

Merge remote-tracking branch 'origin/breaking' into mhauru/delete-flags

4f85f2b

mhauru requested a review from penelopeysm September 30, 2025 16:19

penelopeysm reviewed Sep 30, 2025

View reviewed changes

src/varnamedvector.jl Show resolved Hide resolved

penelopeysm reviewed Sep 30, 2025

View reviewed changes

mhauru added 2 commits October 9, 2025 13:53

Rename islinked to is_transformed, remove duplication

0103a97

Merge remote-tracking branch 'origin/breaking' into mhauru/delete-flags

db64645

mhauru requested a review from penelopeysm October 9, 2025 15:18

penelopeysm requested changes Oct 9, 2025

View reviewed changes

HISTORY.md Show resolved Hide resolved

mhauru merged commit 01bf0bc into breaking Oct 10, 2025
18 of 19 checks passed

mhauru deleted the mhauru/delete-flags branch October 10, 2025 08:55

penelopeysm mentioned this pull request Oct 15, 2025

del "del": remove del flag #982

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Replace `Metadata.flags` with `Metadata.trans` #1060

Replace `Metadata.flags` with `Metadata.trans` #1060

Uh oh!

mhauru commented Sep 29, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Sep 29, 2025 •

edited

Loading

Uh oh!

codecov bot commented Sep 29, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

mhauru commented Sep 30, 2025

Uh oh!

yebai commented Sep 30, 2025 •

edited

Loading

Uh oh!

mhauru commented Sep 30, 2025

Uh oh!

Uh oh!

penelopeysm Sep 30, 2025 •

edited

Loading

Uh oh!

penelopeysm left a comment

Uh oh!

Uh oh!

penelopeysm commented Oct 9, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		islinked(vi::SimpleVarInfo) = istrans(vi)
		islinked(vi::SimpleVarInfo) = is_transformed(vi)

Replace Metadata.flags with Metadata.trans #1060

Replace Metadata.flags with Metadata.trans #1060

Uh oh!

Conversation

mhauru commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before

After

Uh oh!

github-actions bot commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Report for Commit a011dd6

Computer Information

Benchmark Results

Uh oh!

codecov bot commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

mhauru commented Sep 30, 2025

Uh oh!

yebai commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mhauru commented Sep 30, 2025

Uh oh!

Uh oh!

penelopeysm Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

penelopeysm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

penelopeysm commented Oct 9, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Replace `Metadata.flags` with `Metadata.trans` #1060

Replace `Metadata.flags` with `Metadata.trans` #1060

mhauru commented Sep 29, 2025 •

edited

Loading

github-actions bot commented Sep 29, 2025 •

edited

Loading

Benchmark Report for Commit `a011dd6`

codecov bot commented Sep 29, 2025 •

edited

Loading

yebai commented Sep 30, 2025 •

edited

Loading

penelopeysm Sep 30, 2025 •

edited

Loading